Image data processing apparatus, image data processing method, and program

ABSTRACT

Image data are compressed such that they do not visually deteriorate. Nonlinear compression and DPCM compression are performed in series. A compressing unit  41  that uses nonlinear transform inputs image data (L bits per pixel) of each color component. Image data are compressed to M bits (M&lt;L) according to a compression transform tale TB 1  that has been set up according to image information extracted from original image data. The compressing unit  41  performs compression transform having a characteristic similar to a gamma curve characteristic used in a gamma correction disposed downstream of the compressing unit  41 . Image data compressed and transformed to M bits by the nonlinear compressing unit  41  are input to a DPCM compressing unit  42 . The DPCM compressing unit  42  finally compresses input data to N bits (for example, N=10) according to a quantization table TB 2 . The compressed image data are stored to an image memory through a packing section  43.

TECHNICAL FIELD

The present invention relates to an image processing apparatus, inparticular, an image data processing apparatus that processes image datacaptured by an image capturing apparatus such as a digital still cameraor a camera recorder (an apparatus into which a video camera and arecorder are integrated), an image data processing method, and a programthat causes a computer to execute the processing method.

BACKGROUND ART

In an image capturing apparatus such as a digital still camera or acamera recorder, when a shutter is pressed, a still picture is captured.For the captured original image data, internal signal processes such asa captured signal process and an encoding process are performed and thenrecorded for example to a detachable record medium. At this point,before the internal signal processes are performed for the capturedoriginal image data, they are temporarily stored in an image memory. Theimage memory is composed of a DRAM (Dynamic Random Access Memory), anSDRAM (Synchronous DRAM), or the like. As the number of pixels hasincreased in recent years, an image memory having a large storagecapacity has become necessary. Thus, hardware cost and power consumptionhave adversely increased.

Thus, as described in Patent Document 1 (Japanese Patent ApplicationLaid-Open No. 2002-111989), in an image capturing apparatus of therelated art, captured original image data are compressed and stored inan image memory (an original image data buffer memory). Data that havebeen read from the image memory are decompressed and then thedecompressed data are processed as described above. Compressingprocesses for original image data include DPCM process, Huffmanencoding, arithmetic encoding for an original image signal, JPEGlossless encoding process that properly utilizes these processes, anduniversal encoding typified by Ziv-Lempel method.

In the DPCM process, the higher the correlativity of a pixel of interestand adjacent pixels becomes, the higher the compression ratio becomes.Thus, to obtain a high compression ratio, the DPCM process is effectivefor high order bits that have high correlativity, for example sixth toeighth bits of image data of 12 bits of one pixel. However, to do that,the DPCM process is not effective for low order bits that have lowcorrelativity. In the DPCM, if an image that has low correlativity suchas an edge of an image is compressed, the compressed image largelydistorts. In the DPCM, an error propagates.

Compressed data that have been read from the image memory aredecompressed and thereby original image data are obtained. For theoriginal image data, signal processes such as Gamma correction, whitebalance correction, and linear matrix are performed. As a result, aluminance signal and two color difference signals are generated. Thegamma correction is a process of which the camera side reverselycorrects nonlinearity of light emitting characteristics of a Brown tube.In the gamma correction, the ratio of which image data having highluminance is compressed is high. Thus, according to characteristics ofthe gamma correction, a compressing system that uses nonlinear transformof which the ratio of which image data having high luminance iscompressed is high has been proposed.

A compressing process that uses such nonlinear transform can be veryeasily structured. However, since the higher the compression ratio is,the more the information of low order bits is lost. As a result, imagequality deteriorates as in solarization. Thus, it is difficult to obtaina high compression ratio. Solarization means a decrease of a developableconcentration of a photographic emulsion because of excessive exposure.Visually, an image having a small number of quantizing bits, namely arough gradation, is generated. As a result, the image quality of a flatimage remarkably deteriorates.

Thus, in the compressing processes of the related art, compressiondistortions differ in their types. In any compressing process, there isa tendency that the higher the compression ratio is, the more the imagequality deteriorates.

Thus, an object of the present invention is to provide an image dataprocessing apparatus, an image data processing method, and a programthat allow encoding efficiency to be improved so as to improve the imagequality.

DISCLOSURE OF THE INVENTION

To solve the foregoing problem, the present invention is an image dataprocessing apparatus which compresses original image data captured by animaging device, comprising:

means for obtaining signals of same color components of color filters;

first compressing means for performing a first compressing process ofcompressing L bits to M (<L) bits for each pixel of image data separatedinto the same color components;

second compressing means, connected to the first compressing means inseries, for performing a second compressing process, which is differentfrom the first compressing process in characteristics of distortionwhich occurs, of compressing M bits to N (<M) bits for each pixel;

an image memory for storing data compressed by the second compressingmeans;

second decompressing means for performing a second decompressingprocess, which is a reverse process of the second compressing process,of decompressing N bits to M bits for each pixel of data stored in theimage memory;

first decompressing means, connected to the second decompressing meansin series, for performing a first decompressing process, which is areverse process of the first compressing process, of decompressing Mbits to L bits for each pixel of output data of the second decompressingmeans; and

signal processing means for performing a signal process including gammacorrection for image data which are output from the first decompressingmeans.

The present invention is an image data processing method of compressingoriginal image data captured by an imaging device, the method comprisingthe steps of:

extracting signals of same color components of color filters;

performing a first compressing process of compressing L bits to M (<L)bits for each pixel of image data separated into the same colorcomponents;

performing a second compressing process, which is different from thefirst compressing process in characteristics of distortion which occurs,of compressing M bits to N (<M) bits for each pixel, the secondcompressing step being performed after the first compressing step;

storing data compressed at the second compressing step to an imagememory;

performing a second decompressing process, which is a reverse process ofthe second compressing process, of decompressing N bits to M bits foreach pixel of data stored in the image memory;

performing a first decompressing process, which is a reverse process ofthe first compressing process, of decompressing M bits to L bits foreach pixel of output data of the second decompressing step, the firstdecompressing step being preformed after the second decompressing step;and

performing a signal process including gamma correction for image datadecompressed at the first decompressing step.

The present invention is a program which causes a computer to execute animage data processing method, the method comprising the steps of:

extracting signals of same color components of color filters of aimaging device;

performing a first compressing process of compressing L bits to M (<L)bits for each pixel of image data separated into the same colorcomponents;

performing a second compressing process, which is different from thefirst compressing process in characteristics of distortion which occurs,of compressing M bits to N (<M) bits for each pixel, the secondcompressing step being performed after the first compressing step;

storing data compressed at the second compressing step to an imagememory;

performing a second decompressing process, which is a reverse process ofthe second compressing process, of decompressing N bits to M bits foreach pixel of data stored in the image memory;

performing a first decompressing process, which is a reverse process ofthe first compressing process, of decompressing M bits to L bits foreach pixel of output data of the second decompressing step, the firstdecompressing step being preformed after the second decompressing step;and

performing a signal process including gamma correction for image datadecompressed at the first decompressing step.

According to the present invention, by combining a plurality ofcompression methods that differ in characteristics, a compressiondistortion is dispersed to noises having different characteristics.Since a visual deterioration is suppressed, a high compression ratio canbe accomplished.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing the structure of an image capturingapparatus according to an embodiment of the present invention;

FIG. 2 is a block diagram showing an example of an image processingsection according to the embodiment of the present invention;

FIG. 3 is a block diagram showing an example of the structure of acompressing/decompressing unit according to the embodiment of thepresent invention;

FIG. 4 is a schematic diagram describing packing to a data bus width ofan image memory;

FIG. 5 is a block diagram showing an example of the structure of acompressing unit that uses nonlinear transform;

FIG. 6 is a block diagram showing an example of the structure of adecompressing unit;

FIG. 7 is a schematic diagram describing a transform curve of thecompressing unit that uses nonlinear transform;

FIG. 8 is a schematic diagram showing an example of which the transformcurve of the compressing unit that uses nonlinear transform isrepresented by a broken line curve;

FIG. 9 is a block diagram showing an example of the structure of a DPCMcompressing/decompressing unit according to the embodiment of thepresent invention;

FIG. 10A, FIG. 10B, and FIG. 10C are schematic diagrams showing examplesof histograms of level distributions of images used to selectquantization tables of the DPCM compressing unit according to theembodiment of the present invention;

FIG. 11 is a schematic diagram showing an example of a quantizationtable of the DPCM compressing unit according to the embodiment of thepresent invention;

FIG. 12 is a schematic diagram showing an example of which aquantization table of the DPCM compressing unit according to theembodiment of the present invention is represented by a broken linecurve;

FIG. 13 is a block diagram showing another example of the structure ofthe compressing/decompressing unit of the image processing section;

FIG. 14 is a block diagram showing an example of an ADRC compressingunit; and

FIG. 15 is a block diagram showing an example of an ADRC decompressingunit.

BEST MODES FOR CARRYING OUT THE INVENTION

Next, with reference to the accompanying drawings, an embodiment of thepresent invention will be described. FIG. 1 shows an example of thestructure of an image capturing apparatus 1 according to the embodimentof the present invention. The image capturing apparatus 1 has a lenssection 2, an imaging device 3, a timing generating section 4, afront-end 5, an image processing section 6, an image memory 7, an imagemonitor 8, an external storage medium 9, and a camera controllingmicrocomputer 10.

The lens section 2 collects light from a subject. The lens section 2includes a lens 11, a diaphragm 12, and a shutter 13. The diaphragm 12controls the amount of light. The shutter 13 blocks the passage of lightso as to control exposure. The diaphragm 12 may also have a function ofthe shutter 13. The diaphragm 12 and the shutter 13 are controlled bythe camera controlling microcomputer 10.

The imaging device 3 is an image sensor such as a CCD (Charge CoupledDevice) or a CMOS sensor (Complementary Metal Oxide Semiconductorsensor) and converts light information of the subject into an electricsignal. Disposed on the front surface of the sensor are a plurality ofcolor filters such as three primary color filters or complementaryfilters. The timing generating section 4 drives the imaging device 3.The timing generating section 4 also controls exposure of high speed-lowspeed electronic shutters, and so forth. The timing generating section 4is controlled by the camera controlling microcomputer 10.

The front-end 5 converts an analog signal that is output from theimaging device 3 into a digital signal. The front-end 5 performsprocesses such as correlatively dual sampling that removes a noisecomponent from an image signal captured by the imaging device 3 andobtains the captured image signal, gain control that controls the levelof the captured image signal, and A/D conversion that converts an analogsignal into a digital signal. The front-end 5 is controlled by thecamera controlling microcomputer 10.

The image processing section 6 performs various types of digital signalprocesses according to captured image data captured by the imagingdevice 3 and converted into the digital signal so as to generate aluminance signal and color signals. The image processing section 6 has afunction of encoding image data into a predetermined file format such asJPEG (Joint Photographic Experts Group).

The image memory 7 is a storage device that temporarily stores imagedata for signal processes of the image processing section 6. The imagememory 7 is for example a DRAM (Dynamic Random Access Memory), an SDRAM(Synchronous DRAM), or the like. When the image memory 7 stores originalimage data supplied from the front-end 5, data compressed by the imageprocessing section 6 are stored in the image memory 7 so as to save thememory capacity.

The image monitor 8 is a monitor on which the user checks an imagecaptured by the imaging device 3 (this image is referred to as athrough-image). In addition, the image monitor 8 displays a menu onwhich the user operates the image capturing apparatus. Moreover, theimage monitor 8 displays reproduced image data. The image monitor 8 isfor example an LCD panel (Liquid Crystal Display Panel). The externalstorage medium 9 is a record medium that stores image data. The externalstorage medium 9 may be typified by a flash memory as a rewritablenonvolatile memory.

The camera controlling microcomputer 10 controls the entire imagecapturing apparatus 1. In other words, the camera control microcomputer10 performs exposure control of the diaphragm 12, open/close control ofthe shutter 13, electronic shutter control of the timing generatingsection 4, gain control of the front-end 5, and mode and parametercontrols of the image processing section 6.

FIG. 2 shows an example of the structure of the image processing section6 according to the embodiment of the present invention. The imageprocessing section 6 has a signal processing section 21, a detectingsection 22, a compressing section 23, a decompressing section 24, amemory controller 25, a memory interface 26, a monitor interface 27, anda microcomputer interface 28. These sections are mutually connected by adata bus 29 and a control bus 30. In FIG. 2, the flow of image data isrepresented by solid lines and the flow of control data is representedby broken lines.

The signal processing section 21 corrects the original image information(raw data) digitized by the front-end 5 with respect to the imagingdevice 3. Namely, the signal processing section 21 corrects defects ofthe original image information. The compressing section 23 compressesoriginal image data that have been corrected and writes the compresseddata to the image memory 7 through the memory controller 25 and thememory interface 26. The compressed data are read from the image memory7 and decompressed by the decompressing section 24. The decompressingsection 24 outputs original image data.

The original image data that have been read from the image memory 7 andthat have been decompressed are supplied to the signal processingsection 21. The signal processing section 21 performs digital signalprocesses such as digital clamp, white balance, gamma correction,interpolation calculation, filter calculation, matrix calculation,luminance generation calculation, and color generation calculation so asto generate an image signal composed of a luminance signal and colordifference signals. The signal processing section 21 generates imagedata encoded according to a predetermined file format such as JPEG.

The present invention can be also applied to a structure that performssignal processes for original image data, compresses an image signalcomposed of a luminance signal and color difference signals and writesthe compressed image signal to the image memory. In addition, as withthe image memory 7, compressed image data may be stored to the externalstorage medium 9 as well as the image memory 7.

The detecting section 22 performs a detecting process of controlling thecamera for the captured image. Detected signals of the detecting section22 are for example a detected signal with respect to auto focus, adetected signal with respect to automatic exposure control, and soforth. The detecting section 22 detects an edge component of theluminance of an auto focus detection area that has been set up at apredetermined position of the captured image as a detected signal withrespect to auto focus, detects the edge component, cumulates the edgecomponents, and outputs a contrast value as the cumulated edgecomponents. In addition, the detecting section 22 detects the luminanceof the detection area of the luminance that has been set up at thepredetermined position on the captured image as a detected signal withrespect to automatic exposure control and outputs the luminance level.

When a still image is captured, the compressing section 23 compresses acaptured image of each color supplied from the front-end 5. The memoryinterface 26 packs the compressed image data with the bus width of theimage memory 7. The packed data are temporarily stored in the imagememory 7 through the memory interface 26. On the other hand, compresseddata are read from the image memory 7 to the memory interface 26. Thememory interface 26 depacks the compressed data. Thereafter, thedecompressing section 24 decompresses the depacked image data for signalprocesses of the signal processing section 21.

The memory controller 25 exchanges image data between sections of theimage processing section 6 and between each section of the imageprocessing section 6 and the image memory 7 and controls the data bus29. The memory interface 26 exchanges image data and compressed datawith the image memory 7 for signal processes of the image processingsection 6. The monitor interface 27 converts image data into varioustypes of display formats for the image monitor 8. For example, an NTSCencoder that displays the image data on an NTSC monitor is known. Themicrocomputer interface 28 exchanges control data and image data betweenthe camera controlling microcomputer 10 and the image processing section6 that the camera control microcomputer 10 controls.

FIG. 3 shows the compressing section 23, the decompressing section 24,the memory controller 25, and the memory interface 26 of the imageprocessing section 6 as functional structures. This image processingapparatus performs a first compressing process and a second compressingprocess that are different processes. The image processing apparatusperforms a nonlinear process as the first compressing process. The imageprocessing apparatus performs DPCM compression as the second compressingprocess. An image distortion that occurs in the first compressingprocess differs from an image distortion that occurs in the secondcompressing process. It appears that image distortions depend oncompressing systems. Thus, there may be various types of pairs ofcompressing systems that can be combined in series. For example,compressing circuits such as nonlinear compression and ADRC (AdaptiveDynamic Range Coding) can be combined. Instead, three or more types ofcompressing systems may be combined.

Next, with reference to FIG. 3, a process of compressing L bits (forexample, L=14) of a still image to N (N<L) bits (for example N=10) willbe described. Since the compressing operation and the decompressingoperation are not simultaneously performed, the structure shown in FIG.3 is shared thereby.

A horizontal synchronous signal, a vertical synchronous signal, anenable signal, and so forth of the image data are input to a timinggenerating unit 40. The timing generating unit 40 generates a timingsignal and a control signal for compressing units 41 and 42,decompressing units 46 and 47, a packing section 43, a depacking section45, the image memory 7, and so forth.

Image data (L bits per pixel) of color components of three primary colorsignals are input to the compressing unit 41 that uses nonlineartransform. Luminance information is extracted for example as imageinformation PI1 from the original image data supplied from the front-end5. The information PI1 is output to the microcomputer 10. Themicrocomputer 10 identifies a characteristic of the input image from theinformation and sets up a compression transform table TB1 suitable forthe information. The compression transform table TB1 that has been setup is fed back to the compressing unit 41 that uses nonlinear transform.The image information PI1 may be information for automatic exposurecontrol obtained by the detecting section 22.

The compressing unit 41 that uses nonlinear transform compresses theimage data to M bits (M<L), for example M=12, according to thecompression transform table TB1. The compressing unit 41 performscompression and transform with characteristics similar to gamma curvecharacteristics of gamma correction of the signal processing section 21(see FIG. 1) disposed on the next stage of the compressing unit 41 so asto weight image data in the same manner as does the signal processingsection 21.

Image data that have been compressed and transformed to M bits bynonlinear transform of the compressing unit 41 are input to the DPCMcompressing unit 42. The DPCM compressing unit 42 compresses thetransformed image data to N bits (for example, N=10) according to aquantization table TB2. Information PI2 is generated to set up aquantization table TB2. The compressed image data are input to thepacking section 43. The compressed image data are packed with the buswidth of the image memory and then stored in the image memory 7. FIG. 4shows an example of packed image data in the case that the path width is16 bits. In this embodiment, the compression ratio of the compressingunit 41 that uses nonlinear transform is ¼ and the compression ratio ofthe DPCM compressing unit 42 is ¼. As a result, a compression ratio of1/16 is accomplished.

Next, with reference to FIG. 3, a process of decompressing N bits to Lbits will be described. The decompressing process is a process performedin the reverse order of the foregoing compressing process. As in thecompressing process, the decompressing process is performed for eachcolor signal of three primary color components. Image data stored in theimage memory 7 are read and input to the depacking section 45.Compressed data multiplexed with the bus width are depacked to N bits ofimage data by the depacking section 45. The depacked image data areinput to the decompressing unit 46 as a second decompressing means.

The DPCM decompressing unit 46 decompresses N bits to M bits accordingto a reverse transform table TB12. The decompressed image data are inputto the decompressing unit 47 that uses nonlinear transform as a firstdecompressing means. The decompressing unit 47 decompresses the imagedata to L bits as original bits according to a decompression table TB11paired with the compression transform table TB1 selected by thecompressing unit 41. The decompressing unit 47 that uses nonlineartransform supplies L bits of original image data of each color componentto the signal processing section 21 (see FIG. 2). The signal processingsection 21 generates an image signal composed of a luminance signal andcolor difference signals by digital signal processes such as digitalclamp, white balance, gamma correction, interpolation calculation,filter calculation, matrix calculation, luminance generationcalculation, and color generation calculation.

FIG. 5 shows a structure that uses nonlinear transform to compress data.Image data of a particular color are supplied to the compressing unit 41and a luminance extracting section 53. The luminance extracting section53 extracts the image information PI1 corresponding to luminance fromoriginal image data supplied from the front-end 5. The extracted imageinformation PI1 is supplied to a compression transform rule settingsection 51.

The compression transform rule setting section 51 sets up a compressiontransform rule to a compression transform table 52 according to theextracted luminance information PI1. In other words, characteristics ofa compression transform table are set up according to a compressiontransform rule. A compression transform table TB1 that has been set upis supplied to the compressing unit 41. The compressing unit 41 performsa compressing process that uses nonlinear transform according to thecompression transform table TB1. M bits of data that have beencompressed are supplied to the next staged DPCM compressing unit 42disposed on the next stage of the compressing unit 41. Information thatidentifies characteristics of the compression transform table that isused is associated with compressed data that are transmitted. When thecompressed data are decompressed, the information is used to represent adecompression transform table. This information may be stored in themicrocomputer 10.

FIG. 6 shows a structure that uses nonlinear transform to decompressdata. Data that have been read from the image memory 7 and decompressedto M bits by the DPCM decompressing unit 46 are supplied to a transformrule setting section 54 and the decompressing unit 46 that usesnonlinear transform. The transform rule setting section 54 outputs adecompression transform table TB11 paired with a compression transformtable used in a compressing process from a decompression transform table55. The decompressing unit 46 decompresses data to N bits according tothe decompression transform table TB11.

FIG. 7 shows examples of curves according to the compression transformtable TB1. The compression transform table TB1 is composed of a set ofcompression pairs of pre-compressed data and post-compressed data. Thecompressing unit 41 compresses data with reference the conversion pairsof the compression transform table TB1. The compression transform tableTB1 is composed of a fixed region and a variable region. In the fixedregion, each transform pair is fixed. In the variable region, thecompression transform rule setting section 51 can change conversionpairs.

In the example shown in FIG. 7, there are three types of transformcurves 61, 62, and 63 in the variable region. These curves are selectedaccording to the luminance information PI1 extracted by the luminanceextracting section 53. When luminance information is extracted for eachframe, transform curves are changed for each frame. When the extractedluminance is low, the transform curve 61 is used. When the extractedluminance is intermediate, the transform curve 62 is used. When theextracted luminance is high, the transform curve 63 is used. It ispreferred that these transform curves 61, 62, and 63 be similar to gammacorrection curves used in the signal processing section 21.

As shown in FIG. 8, the compression transform table TB1 is approximatedby broken line curves as shown in FIG. 8. For example, a broken linecurve is defined by threshold values TH0, TH1, TH2, and TH3 and offsetsOFT0, OFT1, OFT2, and OFT3. In other words, (TH0, OFT0) defines astraight line 64 a. (TH1, OFT1) defines a straight line 64 b. (TH2,OFT2) defines a straight line 64 c. (TH3, OFT3) defines a straight line64 d. 2¹⁴ as the maximum value of the pre-compressed data and 2¹² as themaximum value of the post-compressed data define a straight line 64 e.The larger the suffix of the straight line is, the smaller the slope ofthe straight line becomes. For example, the straight lines 64 a, 64 b,and 64 c are fixed, whereas the straight lines 64 d and 64 e arevariable.

Pre-compressed data are compared with each threshold value and one ofranges of the five lines 64 a to 64 e in which the pre-compressed dataare contained is decided. Since each straight line is represented as alinear function, the value of post-compressed data on each straight linecan be obtained by a linear interpolation. In addition, by changing athreshold value or an offset, characteristics of the compressiontransform table can be changed according to the extracted luminanceinformation PI1. When the offset OFT3 is changed to the offset OFT4, thestraight lines 64 d and 64 e can be changed to the straight lines 65 dand 65 e, respectively.

It is not necessary to divide a transform curve into a fixed region anda variable region. Instead, a whole curve may be variable. In addition,values other than offsets as representative values may be obtained bynon-linear interpolations instead of linear interpolations.

FIG. 9 shows the DPCM compressing units 42 and 46 that share a circuitfor a compressing operation and a circuit for a decompressing operation.The compressing unit 41 that uses nonlinear transform supplies inputimage data of which L bits, for example 14 bits, have been compressed toM bits, for example 12 bits, to a subtracting unit 70. The subtractingunit 70 obtains a prediction error of a prediction value predicted froma past pixel defined with a delay element D by a predicting unit 71, forexample a pixel on the same line as and adjacent to a pixel of interest.In the predicting unit 71, “a” represents a weighting coefficient withwhich a prediction value is generated.

The prediction error is input to a difference histogram detecting unit72, a quantizing unit 73, and a quantizing/dequantizing unit 74. Thequantizing/dequantizing unit 74 has a structure that simultaneouslyperforms a quantizing process and a dequantizing process of transforminga quantized value into a representative value. Thequantizing/dequantizing unit 74 has a quantizing characteristic that isthe same as a quantizing characteristic that the quantizing unit 73 hasand a dequantizing characteristic that a dequantizing unit 75 has.Output data of the quantizing/dequantizing unit 74 are supplied to anadding unit 77 through a selector 76.

When data are compressed, the selector 76 selects an input terminal “a”according to a compression/decompression switching signal SC. When dataare decompressed, the selector 76 selects an input terminal “b”according to the compression/decompression switching signal SC. Whendata are compressed, data of which an output of thequantizing/dequantizing unit 74 and an output of the predicting unit 71are added by the adding unit 77 are supplied to the subtracting unit 70.The subtracting unit 70 calculates a prediction error. By disposing aquantizing unit in a feedback loop and providing a circuit having thesame structure as the decompressing unit, a quantizing noise that occursin the quantizing unit can be prevented from cumulating in thedecompressing unit.

The difference histogram detecting unit 72 generates a histogram thatrepresents occurrence frequencies of prediction errors of one capturedstill image and outputs the values as the image information PI2. Itappears that a histogram obtained by the difference histogram detectingunit 72 largely deviates depending on an image signal. The quantizationtable TB2 used in the quantizing unit 73 and the quantizing/dequantizingunit 74 is adaptively changed according to the detected histogram so asto effectively compress data. When data are compressed, compressed dataof which data have been compressed to N bits, for example 10 bits, areobtained from the quantizing unit 73. The obtained compressed data arewritten to the image memory 7.

Data that have been read from the image memory 7 are supplied to thedequantizing unit 75. The dequantizing unit 75 decompresses N bits to Mbits according to the reverse transform table TB12 compared with thequantization table TB2 used in the quantizing unit 73. Information thatrepresents a reverse transform table to be selected is stored in theimage memory 7. Instead, a reverse transform table may be set upaccording to information supplied from the microcomputer 10.

A prediction error that has been decompressed to M bits is supplied fromthe dequantizing unit 75 to the adding unit 77 through the selector 76.A prediction value generated by the predicting unit 71 is fed back tothe adding unit 77 and a restored value is obtained from the adding unit77. The restored value is supplied to the decompressing unit 47 thatuses nonlinear transform.

FIG. 10A, FIG. 10B, and FIG. 10C show schematic examples of histogramsthat represent occurrence frequencies of prediction errors obtained asan output of the subtracting unit 70. In these drawings, the horizontalaxis and the vertical axis represent prediction errors and frequencies,respectively. Prediction errors increase in the direction of thehorizontal axis. Instead, the range of the minimum value and the maximumvalue of prediction errors may be divided by a predetermined number andthe occurrence frequencies of each of the divided ranges may bedetected. For example, occurrence frequencies of prediction errors ofone image are obtained. A histogram is generated according to theobtained occurrence frequencies. The quantization table TB2 used fornonlinear compression of the quantizing unit 73 and thequantizing/dequantizing unit 74 is set up according to the generatedhistogram.

FIG. 11 shows an example of the quantization table TB2 where thehorizontal axis and the vertical axis represent prediction errors andrepresentative values, respectively. Reference numeral 81 represents astandard quantization table. A quantization table 82 and a quantization83 are set up to the quantization table 81. In these quantizationtables, based on the fact that the sensitivity of human eyes is higherfor a portion having a large difference than for a portion having asmall difference, the compression ratio of a portion having a largeprediction error is caused to be higher than the compression ratio of aportion having a small prediction error.

As shown in FIG. 10A, the quantization table 82 is set up to an image ofwhich prediction errors are distributed in a region of small values,namely a flat image. In the quantization table 82, representative valuesthat are different are output for prediction errors in the range ofrelatively small levels from 0 to threshold value A. In contrast, in thequantization table 82, only a maximum representative value is output forprediction errors in the range of levels that exceed threshold value A.In other words, many bits are assigned to low levels of predictionerrors. In this example, the number of bits that occur in one image suchas one frame is a predetermined value or less. Thus, to decrease thecompression distortion, it is necessary to decide the numbers of bitsassigned to prediction errors.

As shown in FIG. 10B, the quantization table 81 is set up to a normalimage of which prediction errors are relatively evenly distributed inthe range of small levels to some extent. In the quantization table 81,representative values that are different are output for predictionerrors in the range of levels from 0 to threshold value B (>A). Incontrast, in the quantization table 81, only a maximum representativevalue is output for prediction errors in the range of levels that exceedthreshold value B. In other words, many bits are assigned to low levelsto intermediate levels of prediction errors.

As shown in FIG. 10C, the quantization table 83 is set up to an image ofwhich prediction errors are distributed in the range of large levels,namely an image in which adjacent pixels have less correlation with eachother because of a fine image pattern. In the quantization table 83,representative values that are different are output for predictionerrors in the range of levels from 0 to threshold value C (>B). In thequantization table 83, only a maximum representative value is output forprediction errors in the range of levels that exceed C. In other words,bits are assigned to prediction errors in the range from small levels tolarge levels.

As described above, a quantization table is set up corresponding to thedistribution of levels of prediction errors of one captured stillpicture. As a result, quantization can be performed with smallcompression distortion.

As shown in FIG. 12, in reality, a quantization table can beapproximated by characteristics of broken lines. In FIG. 12, thevertical axis and the horizontal axis represent prediction errors andrepresentative values, respectively. Threshold values th0 to th6 aredefined for prediction errors. Offsets oft0 to oft5 are defined forrepresentative values. Straight lines defined by these threshold valuesand offsets have slopes such that transform coefficients multiplied byprediction errors become (1, ½, ¼, ⅛, 1/16, 1/32, and 1/64).

Data of pairs of threshold values and offset values have been stored inthe memory. Threshold values and prediction errors are compared andstraight lines in which prediction errors are contained are decided. Asa result, representative values corresponding to the prediction errorsare obtained. In this case, representative values are obtained bycalculating linear interpolations. By changing at least one of athreshold value and an offset value, characteristics of a quantizationtable can be changed.

Thus, according to this embodiment of the present invention, since twodifferent compressing means of nonlinear transform compression using avisual characteristic that the sensitivity of human eyes is high for alow gradation portion and DPCM compression using both differencesensitivity that the sensitivity of human eyes is high for a portionhaving small difference and correlation of an image signal are provided,a quantizing error is dispersed such that the user does not become awareof noises. In addition, the nonlinear compression/decompressiontransform table is adaptively changed such that bits are assignedaccording to a histogram. Moreover, a quantization table is optimallychanged according to a histogram distribution of prediction errors. As aresult, representative values are assigned according to a distributionof prediction errors.

As a result, a compression ratio can be increased, while visibility ofnoises does not change after signal processes are performed. With anincrease of the compression ratio, the number of still images stored inthe image memory can be increased. In addition, with a decreasedfrequency band in which the image memory is accessed and decreased powerconsumption, the service lives of batteries of the digital still cameraand camera recorder can be more prolonged.

FIG. 13 shows another embodiment of the present invention. In thisembodiment, as two compression transforming processes having differentcharacteristics, nonlinear compression and ADRC (Adaptive Dynamic RangeCoding) are combined. With reference to FIG. 13, a process ofcompressing L bits, for example L=14, of a still image to N (N<L) bits,for example N=10, will be described. Since a compressing operation and adecompressing operation are not simultaneously performed, a circuitshown in FIG. 13 can be shared by the compressing operation and thedecompressing operation.

A horizontal synchronous signal, a vertical synchronous signal, anenable signal, and so forth of image data are input to a timinggenerating unit 90. The timing generating unit 90 generates a timingsignal and a control signal for compressing units 91 and 92,decompressing units 96 and 97, a packing section 93, a depacking section95, the image memory 7, and so forth.

The compressing unit 91 that uses nonlinear transform is the same as thecompressing unit 41 of the embodiment shown in FIG. 3. In other words,image data (L bits per pixel) of color components of three-color signalsare input to the compressing unit 91. The compressing unit 91 extractsluminance information from image information PI3, for example originalimage data supplied from the front-end 5. The microcomputer 10identifies a characteristic of the input image from the imageinformation PI3. The microcomputer 10 supplies a compression transformtable TB3 that has been set up according to the image information PI3 tothe compressing unit 91 that uses nonlinear transform.

The compressing unit 91 that uses nonlinear transform compresses imagedata to M bits (M<L), for example M=12, according to the compressiontransform table TB3. A transform curve with which the compressing unit91 performs nonlinear transform has a characteristic that is the samethat of a gamma correction curve used in gamma correction of the signalprocessing section 21 (see FIG. 1) disposed downstream of thecompressing unit 91.

The compressing unit 91 inputs image data, of which L bits have beencompressed and transformed to M bits by nonlinear transform, to the ADRCcompressing unit 92. The ADRC compressing unit 92 outputs image datathat have been compressed finally to N bits (for example, N=10)according to a quantization table TB4 that has been set up according toimage information PI4. The compressed image data are input to thepacking section 93. After the compressed image data are packed with thebus width of the image memory 7, the image data are stored in the imagememory 7.

Next, with reference to FIG. 13, a process of decompressing N bits to Lbits will be described. The decompressing process is performed in thereveres order of the foregoing compressing process. As in thecompressing process, the decompressing process is performed forindividual color signals of three primary color components. Image datastored in the image memory 7 are read and input to the depacking section95. Compressed data multiplexed with the bus width are restored to Nbits of image data by the depacking section 95 and then input to theADRC decompressing unit 96.

The ADRC decompressing unit 96 decompresses N bits to M bits accordingto a reverse transform table TB14 paired with the quantization tableTB4. The decompressed image data are input to the decompressing unit 97that uses nonlinear transform. The decompressing unit 97 decompressesthe image data to the original number of bits, namely L bits, accordingto a decompression table TB13 paired with the compression transformtable TB3 selected by the compressing unit 91. Original image data of Lbits of each color component are supplied from the decompressing unit 97that uses nonlinear transform to the signal processing section 21 (seeFIG. 2). The signal processing section 21 performs digital signalprocesses such as digital clamp, white balance, gamma correction,interpolation calculation, filter calculation, matrix calculation,luminance generation calculation, and color generation calculation so asto generate an image signal composed of a luminance signal and colordifference signals.

The compressing unit 91 that uses nonlinear transform can have thestructure shown in FIG. 5, whereas the decompressing unit 97 that usesnonlinear transform can have the structure shown in FIG. 6.

Since a plurality of pixels that are spatially or chronologicallyadjacent have much correlation with each other, in ADRC, image data arecompressed in the direction of levels of pixels. FIG. 14 shows anexample of the compressing unit 91. Pixel data of which each pixel has Mbits (data of one color component) are supplied to a block segmentingcircuit 101. The block segmenting circuit 101 segments the pixel datainto blocks of a two-dimensional region composed of a plurality ofpixels. A dynamic range (DR) detecting circuit 102 detects a maximumvalue MAX and a minimum value MIN of each block and detects a dynamicrange DR as a result of MAX-MIN.

A subtracting unit 103 subtracts the minimum value MIN from the value ofeach pixel. Data supplied to the subtracting unit 103 may be delayeduntil the data have been detected by the detecting circuit 102. Thesubtracting unit 103 normalizes data of each block. Output data of thesubtracting unit 103 are supplied to a quantizing unit 104.

The quantizing unit 104 performs quantization using the quantizationtable TB4 and the dynamic range DR and outputs a code DT of N bits. Inlinear quantization, a quantizing step Δ to which the dynamic range DRis multiplied by ½^(N) is generated and data excluding a minimum valueis divided by the quantizing step Δ. In this example, the quantizingunit 104 performs nonlinear quantization according to the quantizationtable TB4 that is the same as the quantization table of the DPCMcompressing unit 42 of the foregoing embodiment and changes thequantizing characteristic according to a characteristic of the image,for example a histogram of a level distribution of the foregoingembodiment.

For example, the quantizing step Δ is not constant. Instead, thequantizing step Δ is changed in a plurality of ranges into which thelevels of data excluding the minimum value are divided. In other words,in the range of which the levels of data excluding the minimum value aresmall, the quantizing step is decreased. In the range of which thelevels are large, the quantizing step is increased. Thus, the quantizingstep is not constant. Instead, the quantizing step is changed accordingto a histogram of a level distribution.

Information (not shown) that represents the dynamic range DR, theminimum value MIN, the code DT, and the quantization table TB4 arepacked by the packing section 93 and written to the image memory 7. Theinformation that represents the quantization table TB4 may be stored inthe microcomputer 10.

Data that have been read from the image memory 7 are depacked by thedepacking section 95 and the dynamic range DR and the code DT aresupplied to a dequantizing unit 112. The reverse transform table TB14paired with the quantization table TB4 is also supplied to thedequantizing unit 112. The reverse transform table TB14 is identifiedaccording to information that denotes that quantization has beenperformed according to the quantization table TB4.

The dequantizing unit 112 transforms the code DT to a representativevalue according to the reverse transform table TB14. In linearquantization, the quantizing step Δ is obtained from the dynamic rangeDR. By multiplying the value of the code DT by the quantizing step Δ, arepresentative value is obtained. With the quantizing step Δ definedaccording to the reverse transform table TB14, a representative value iscalculated.

In the ADRC process, each block is independently quantized. Thus, when ablock has much correlation, namely the dynamic range DR is small, a highcompression ratio can be obtained. In contrast, when a block does nothave much correlation, namely the dynamic range DR is large, there is apossibility that different quantization between blocks results in ablock distortion. On the other hand, in the compressing process thatuses nonlinear transform, although the structure can be simplified, thehigher the compression ratio becomes, the more the information of lowerbits is lost. As a result, image quality deteriorates as insolarization.

In this embodiment of the present invention, since compressing processeshaving different compression characteristics are performed in series, ahigher compression ratio can be obtained than otherwise. In addition,compression distortion is dispersed to noises having differentcharacteristics. Thus, visible deterioration of the image quality can besuppressed.

The present invention is not limited to the foregoing embodiments.Instead, various modifications and ramifications of these embodimentsmay be made. According to the present invention, the number of bits thateach compressing unit connected in series compresses is fixed. Forexample, a first-staged compressing unit compresses L bits to M bits. Asecond-staged compressing unit compresses M bits to N bits. Instead,(L-N) bits to be compressed and assigned to each compressing unit may bechanged according to a characteristic of an image to be processed.

In addition, compressing/decompressing units are provided for individualcolor components of color signals. However, to reduce the circuit scale,all color components may be multiplexed and processed by a time divisionprocess. In addition, the present invention can be applied to the caseof which an image sensor has color filters of four or more colors. Inaddition, according to the present invention, the same number of bitsare assigned to each color component. Instead, by changing the number ofbits assigned to each color component, namely changing the compressionratio depending on deviation of arrangement of color components, ahigher compression ratio is expected to be obtained. In addition,different number of bits may be assigned to color components dependingon weights of color components in signal processes performed ondownstream stages and an arrangement condition of color filters.

In addition, the present invention can be applied not only to the caseof which a still image that is being monitored is captured, but also tothe case of which a still image is captured while a moving image isbeing recorded.

The processing means according to the foregoing embodiments may not be amethod having a sequence of steps. Instead, the processing means may bea program that causes a computer to execute a sequence of means or arecord medium on which the program is stored.

DESCRIPTION OF REFERENCE NUMERALS

-   -   1 IMAGE CAPTURING APPARATUS    -   3 IMAGING DEVICE    -   6 IMAGE PROCESSING SECTION    -   7 IMAGE MEMORY    -   10 MICROCOMPUTER    -   41, 91 COMPRESSING UNIT THAT USES NONLINEAR TRANSFORM    -   42 DPCM COMPRESSING UNIT    -   46 DPCM DECOMPRESSING UNIT    -   47, 97 DECOMPRESSING UNIT THAT USES NONLINEAR TRANSFORM    -   92 ADRC COMPRESSING UNIT    -   96 ADRC DECOMPRESSING UNIT

1. An image data processing apparatus which compresses original imagedata captured by an imaging device, comprising: means for obtainingsignals of same color components of color filters; first compressingmeans for performing a first compressing process of compressing L bitsto M (<L) bits for each pixel of image data separated into the samecolor components; second compressing means, connected to the firstcompressing means in series, for performing a second compressingprocess, which is different from the first compressing process incharacteristics of distortion which occurs, of compressing M bits to N(<M) bits for each pixel, the second compressing means including aquantizing/dequantizing unit which simultaneously performs a quantizingprocess and a dequantizing process within a feedback loop; an imagememory for storing data compressed by the second compressing means;second decompressing means for performing a second decompressingprocess, which is a reverse process of the second compressing process,of decompressing N bits to M bits for each pixel of data stored in theimage memory, the second decompressing means including a dequantizingunit having a dequantizing characteristic that is the same as adequantizing characteristic of the quantizing/dequantizing unit; firstdecompressing means, connected to the second decompressing means inseries, for performing a first decompressing process, which is a reverseprocess of the first compressing process, of decompressing M bits to Lbits for each pixel of output data of the second decompressing means;and signal processing means for performing a signal process includinggamma correction for image data which are output from the firstdecompressing means, the feedback loop including a predicting unit thatis shared by the second compressing means and the second decompressingmeans.
 2. The image data processing apparatus as set forth in claim 1,further comprising: first and second image information extracting meansfor extracting image information necessary to decide compressiontransform rules suitable for an image and a compression method when thefirst and second compressing means perform their compressing processes,respectively; compression transform rule setting means for setting upthe compression transform rules for the first and second compressingmeans according to the extracted image information; and decompressiontransform rule setting means for setting up decompression transformrules for the first and second decompressing means, respectively.
 3. Theimage data processing apparatus as set forth in claim 2, wherein thecompression transform rules are transform characteristics for the firstand second compressing means, respectively, and wherein thedecompression transform rules are reverse transform characteristics forthe first and second decompressing means, respectively.
 4. The imagedata processing apparatus as set forth in claim 2, wherein thecompression transform rules are compression assignments for the firstand second compressing means, respectively, and wherein thedecompression transform rules are decompression assignments for thefirst and second decompressing means, respectively.
 5. An image dataprocessing method of compressing original image data captured by animaging device, the method comprising the steps of: extracting signalsof same color components of color filters; performing a firstcompressing process of compressing L bits to M (<L) bits for each pixelof image data separated into the same color components; performing asecond compressing process, which is different from the firstcompressing process in characteristics of distortion which occurs, ofcompressing M bits to N (<M) bits for each pixel, the second compressingstep being performed after the first compressing step, the secondcompressing process including performing a quantizing within a feedbackloop that includes a quantizing/dequantizing unit which simultaneouslyperforms a quantizing process and a dequantizing process; storing datacompressed at the second compressing step to an image memory; performinga second decompressing process, which is a reverse process of the secondcompressing process, of decompressing N bits to M bits for each pixel ofdata stored in the image memory, the second decompressing processincluding the use of a dequantizing unit having a dequantizingcharacteristic that is the same as a dequantizing characteristic of thequantizing/dequantizing unit; performing a first decompressing process,which is a reverse process of the first compressing process, ofdecompressing M bits to L bits for each pixel of output data of thesecond decompressing step, the first decompressing step being preformedafter the second decompressing step; and performing a signal processincluding gamma correction for image data decompressed at the firstdecompressing step, the feedback loop including a predicting unit thatis shared by the second compressing process and the second decompressingprocess.
 6. A non-transitory computer-readable medium storing a programwhich causes a computer to execute an image data processing method, themethod comprising the steps of: extracting signals of same colorcomponents of color filters of an imaging device; performing a firstcompressing process of compressing L bits to M (<L) bits for each pixelof image data separated into the same color components; performing asecond compressing process, which is different from the firstcompressing process in characteristics of distortion which occurs, ofcompressing M bits to N (<M) bits for each pixel, the second compressingstep being performed after the first compressing step, the secondcompressing process including performing a quantizing within a feedbackloop that includes a quantizing/dequantizing unit which simultaneouslyperforms a quantizing process and a dequantizing process; storing datacompressed at the second compressing step to an image memory; performinga second decompressing process, which is a reverse process of the secondcompressing process, of decompressing N bits to M bits for each pixel ofdata stored in the image memory, the second decompressing processincluding the use of a dequantizing unit having a dequantizingcharacteristic that is the same as a dequantizing characteristic of thequantizing/dequantizing unit; performing a first decompressing process,which is a reverse process of the first compressing process, ofdecompressing M bits to L bits for each pixel of output data of thesecond decompressing step, the first decompressing step being preformedafter the second decompressing step; and performing a signal processincluding gamma correction for image data decompressed at the firstdecompressing step, the feedback loop including a predicting unit thatis shared by the second compressing process and the second decompressingprocess.